tests: add integration test suite for reindexing #28
Conversation
By default, the app now logs at INFO level, without requiring an opt-in. Accordingly, I've dialed down the load frequency on `penumbra-reindexer archive` to log every 10,000th block, rather than every thousand.
Adds integration tests that fetch historical node archives from known URLs, unpack them, and build a `reindexer_archive.bin` containing all historical blocks for the target chain.
There was a problem hiding this comment.
This looks really cool!
The PD crashing thing is really a shame, because it would be nice to use the data structure we have representing what the artifacts look like to also figure out what the two integration test splits need to look like, but there's no good way to encode that, unfortunately.
I think subsequently it would be nice to refactor things a bit so that way pointing at artifacts.plinfra.net isn't hardcoded, but that can be a follow up.
Let's at least look to removing the sys-exit behavior in pd going forward. I'm skeptical it'll be feasible to backport revising the halt behavior for historical versions, but it's on the table.
My thoughts exactly, it was tempting to make things DRY enough to generate plans from the archive declarations, but we're not quite there yet. |
Adds integration tests that fetch historical node archives from known URLs, unpack them, and build a
reindexer_archive.bincontaining all historical blocks for the target chain. Only exercises thepenumbra-reindexer archivebehavior; doesn't yet exercise thepenumbra-reindexer regenfunctionality. Future work can implement that.For now, it's enough to have a reproducible process that confirms all historical chain state (for the popular chains like penumbra-1 and penumbra-testnet-phobos-2) can be collected to form a complete picture of event data. This reproducible process does rely on the existence of external data sources, in the form of historical archives in a known format at known URLs. But at least now we can verify that's the case, with known checksums, as well.
Testing and review
You can pull down this branch and run
just integration; that'll do needful:penumbra-reindexer archiveto create an sqlite3 databaseHowever, be aware that doing so is a disk- and bandwidth-intensive process. On my machine, the process takes ~60-90m, and results in about 500GB of diskspace being used.